Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural processing (NLP) tasks. However, most pretraining efforts focus general domain corpora, newswire and Web. A prevailing assumption is that even domain-specific can benefit by starting from general-domain models. In this paper, we challenge showing for domains with abundant unlabeled text, biomedicine, models scratch results in substantial over continual of To facilitate investigation, compile a comprehensive biomedical NLP benchmark publicly-available datasets. Our experiments show serves solid foundation wide range tasks, leading new state-of-the-art across the board. Further, conducting thorough evaluation modeling choices, both task-specific fine-tuning, discover some common practices are unnecessary BERT using complex tagging schemes named entity recognition (NER). help accelerate research NLP, have released our pretrained community, created leaderboard featuring BLURB (short Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.